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ABSTRACT 

* This is 'a workshop simulation and interpretation 
guide designed for Title I teachers and district personnel, Ihe 
participants stufcu^d have some experience with the aora-referefcced 

^evaluation atfdel. They learn to interpret notmal curve equivalents 

* (NCE> and NCE gains. Participants* are led through an interpretive 
hierarchy fro» simple, descriptive ^interpretations of NCE gains' to 
comparisons of results to pther standards, * ThVs guide gaves £ brief 

'introduction to exploratory/diagnostic and comprehensive M 
interpretations*. The simulation ptovides a* complete evaluation report 
for- a Title * I project, A presenter's guide provides detailed 
information for the workshbp leader- Specific objectives for 

. participants include description of meaning of* NCE gain, conversion 
to a percentile' change, construction of confidence bands around gains 
and use of .then to guide interpretations. Other areas are 
identification of comparisons for ricE^. gains f determination of gain 
difference from a standard, determination cfe whether a. project needs 
more thorough investigation and where to focus it, and identification 
of areas to gather ^^Epr station. (DHH) 
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Syn'opsis ✓ , 

INTERPRETING TITLE I EVALUATION RESULTS 



Rationale ' • 

The purpose is to provide participants with a better understanding of the 
results f^ora 3 riorra-ref ererice<T~e~valuation. These materials help define 
what Interpretations can be "raadfe from the end-of-year evaluation report. 

The two components of the workshop, the simulation and the interpretation 
guide are based on an interpretative hierarchy: I 

1. Descriptive — simple interpretations dealing with what an NCE 

gain signifies. 

2. Comparative — interpretations based on comparing the results to . 

other stand^jpfis 

3. £xploratory/,pf agnostic — interpretations that involve more 

^ thorough descriptions of program Components and , 

achievement impact 

4. * Comprehensive — interpretations that expand beyond the 

^ 1 ' achievement impact such as student attitude effects 

* and the -effects on others besides students. 

The activities in this worfksftop concentrate on the first two levels "w^ith 
a brief Introduction into the third level and a' mention of the fourth. 



Assumptions About the Participants 

The 9 intende^ audience is the Title I teacher. However, District title I 
personnel aftid patents have attended the workshop and found it helpful. 
Familiarity 'with %he norm-referenced evaluation model is the^nost 
important entry confeideration. The audience slfculd have some' experience 
with or" knowledge of the' model and the reporting forms used" with the 
evaluation, if they do not, there is a tendency for the participants to 
get involved more with questions of model implementation and form 
completion than with interpretation. * Therefore, a participant should 
have at legist -one of the following experiences: 

• Completion of a Model A orientaton workshop , 

V 

• Implementation of the evalqation model and completion of report- 
forms , 

• Attendance at an end^o£^ea£«jdafca..analy^^ 
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Participant Objectives ' 

9 

After th is workshop / part icipants should be able to: Al. 

Describe what an NCE gain means * 

* * 

Convert an NCE gain to a percentile change 
Construct a confidence band around an NCE gain 

Us-e confidence bands to guide their interpretations 

* • . ■* 

Identify some- comparisons for an NGjS gam 

Determine' whether an NCE gain is different from a standard 

Determine whether a project needs a more thorough invefetigatibn 

Determine where to focus an investigation of a project 

Identify some areas to gather information 



Instructional Activities (1 1/2 hours) : 

• Introduction to Workshop Components 

• Introduction to Simulation^ 

^ • Discussion of Descriptive Interpretations 

• Discussion of Comparative Interpretations 

• Discussion of - Exploratory/Diagnostic 
Interpretations ^ 



(5 minutes) 
(5 minutes) 
(30 minutes) ♦ 
(30 mintites)^ 

415 minutes) 



Materials Needed 

• Overhead projector and screen , * 

• • Workshop transparencies and blank transparencie^ 

• Workshop materials to distribute to eve*y person 



Description 



y 



The coz^of the workshop is'tfie Interpretation Guide that the participants 
work through with the aid of the simulation activities based pn the' \«* 
end-of-year report from a hypothetical school district. It .is most % ^ 
effective if participants have recent results from their own evaluations * to 
which they can apply the strategies of the workshop. * ^ 
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Component Description- 
INTERPRETING TITLE I EVALUATION RESULTS ^ 

( 

Interpretation Guide for Title I Evaluation Results ' 

This guide is designed to take the participant from simple 
explanations of 'what an NCE gain means to more complicated concerns 
such as where to look for the strengths and weaknesses in the' 
program. To dp this, the guide is divided into three major sections. 

Section One deals with interpretations of NCEs and NCE gainst The 
participant will find explanations' as well as alternative ways of 
describing gains. Also, problems of interpreting gadns for small 
cla^s sizes will be discussed. 

In Secb4ai/Two the participant will start with these initial 
interpretations and add Some information by comparing these results 
to different standards. These standards include last year's results, 
other projects' results, state averages" or objectives set by the 
program. 

Finally, in Section THree the participant will find a discussion of 
whether further investigation is needed based on substantiating the 
results and then where to begin looking for more inforraaticTNtebout a 
program. « 

Interpreting Title I Evaluation Results: Simulation 

A simulation activity designed to take the participants through the 
sections of the Interpretation Guide and to provide exercise in doing 
some of the calculations mentioned in the guide. 

The major ^ections of the simulation are; 

a. Title I Reding Project Description, Wilson Elementary 
School, This Year's Achievement Impact Report 

b. This Year's Achievement Impact Report for Cherry Hill 

Elementary Sciilbol 
Last Year's Achievement Impact Report for Wilson Elementary 
/ School 

Achievement Impact Information from This Year's State 

Title I Report • 
Program Objectives for Centennial School District's 
Title I Reading Project. 

^c. Graphs for showing Wilson's results / ■ 

d. Title I Director's Annual Project Summary 
Tit^e I Evaluator''s Summary 



0 
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Presenter's Guide for the 
Interpreting Title I Evaluation Results Workshop 
Materials Participants Should Have at Outset of workshop - ^ 

1. Interpretation Guid^ for Title I Evaluation Results 

2. Title I Reading Project Description for Wilson Elementary -School, 

3. This year's Achievement Infojpation Report for Wilson Elementary 
School . « 



Introduction to the Workshop 

Introduce P^esejpters. * % 

Tell participants .there were two questions that we've seen in the field 
that initiated the formation of this workshop: "I wonder what this 
end-of-year report information means to me?" and "Last year we got a 7 
NCE gain and this year we only got 6; is something wrong with our 
program?" * * 

This workshop will try to help people address these questions. First, we 
want to provide participants with more information on what the NCE gains 
mean to them and how they pan be used to provide information about a 
prog ram ^Secondly, we w*uld like to help them be able to explain the- 
results- others. This is not only useful to the Title I teacher but 

, that carL also be used in explaining reftults to, other audiences like 
' parents, administrators, or school boards. (This second point? is 

t important to emphasize because it can provide a participant an excuse to 

' ask a question; it Jis much easier to ask "How do I explain this to other 

; teachers?" than it is tcysay "I don't understand it.") 

Materials Strategies 



Explain to the participants that the piece from the 
workshop that we expect them to use after they return 
* to their schools is the Interpretation Guide . Rather 
than simply walking through the guide, we are g'oing to 
use a simulation to explain the different sections of 
the guides 

» 

Handout The brifef description of the Wilson Elementary Reading 

15.207 project is* to give them a feel that the Wilson project 

is not unusual. It 4s not essential that the 
description be read in detail and the aspects of the 
Wilson project be/fhemorized. 

Transparency The first point we should make before we start 

No. l interpreting the Wilson project is that we are 

assuming the information from the Wilson project is 
valid. Page 2 of the Interpretation Guide lists some 
points which we will assume that Wilson has done. 

4 1659D 
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* % \ Materials 



Strategies 

(Note to the presenter, , Oije of the problems with the 
Wilson example is that the test doe$ not match the 
objectives of the project. Violation' of this initial 
assumption will be' discussed later in the simulation.) 



Descriptive tnterpre tatiohs — Simulation 



Handout I-H-I 
Transparency 
Nos. 2, 3 



Have participants *key in on the bottom right hand 
corner of Wilson "Elementary Achievement Information 
form for June 15/ 1980, where 'NCE gains are figured. 
Give them a few seconds to digest tfie info raation. 
there are participants who are unsure about what an 
NCE is /refer them to pages 3 and 4 of the 
Interpretation Guide . Some suggested questions and 
responses are: 

• Did the*Wilson project have an impact? 
On these students, yes. 



If 



Transparency 
Nos. 4,5,6 



Was the impact positive or negative? 
Positive. / 



1 



Should the Wilson teacher be satisfied with these 
results? ^ 

Some teachers will hedge, but invariably someone 
will say "Anything above zera is good." ' • 

Satisfaction is another realm. Teachers will 
compare it to their experiences, but no one can 
really declare that it is satisfactory. 

How would you describe the impact of this project? 
(Page 5 of the Interpretation Guide reviews' the 
assumptions of the norm-referenced evaluation 
model and how impact is. defined.) Page 7 
discusses how to interpret cin NCE gain in terms 
of a percentile change. NCEs have received so 
milch Title I press that we have inadvertently 
implied that percentiles are just as bad as grade 
equivalents. For presenting the effect\of a 
Title I project to teachers, administrators, or 
boards, it might be easier to use percentiles. 
Put the Wilson example in terms of percentile 
gro,wth. Caution participants against comparing 
absolute percentile gains — that's the reason NCEs 
were devised. 



ERLC 



10 



1659D 
2/81 



- Strategies 

• l 

Was the Title I project effective for all ( 
students? „ 

You can't tell from these numbers^. NCE gains 
here refer to the grpup average. Some students 
gained more than this average; some, less? some 
may even have lost.* The NCE gain reflects the 
average. Another thing you might want td look at 
is tlje range of NCE gains' fe>r„ the group of <* 
students. This gives tfou aX idea\of the extremes 
of the gains or losses. Yot raig^t even want to 
make a ^Ls.tograra shoeing' the gains or calculate 
the standard deviation of the gains. Alllpf 
these prbvide more 'information to interpret the 
effectiveness of a project. 

What does the tiCE /gain tell yop afclout the 
effectiveness of /the project for students like 
this? 

Refer participants to^ page 9 of the 
Interpretation Guide . This starts the 
introduction to confidence bands around the NCE^ 
gain. These bands come from two sources pf error: 

f. 

1) Measurement Error . To introduce the 
particfpants t;o the idea of confidence bands 
refer to the score bands with ,which »raost * 
teachers are art least faintly familiar from 
a teacher training measurement course. 

<♦ 

2) Sampling Error . Most teachers . have 
experienced the case where in a small class f 
almost all students do well except for one 
or two who do so poorly they pull down the 
class average to where it doesn't truly 
reflect the project's effect. This kind of 
error is most pronounced in small classes- 
Tell the participants that there are two 
questions that are 'similar but different 
enough to make the answers different. The 
first question is "Was the Title I program ' 
effective for this particular group of 
students?" The^answer . to this is^ obvious 
from the NCE gain on the report. The second 

•question is "Is this Title I program 
effective for other students like* these? If 
I didn't changfe anythirtg for next year/ what 
kind of gains could r expect?" >The answer 
to this question involves the use o'f 
confidence bands. 
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-Strategies ' • 

. 'i 
} Figure 3 oo page^lO of the Interpretation 
Guide shows the relation between class size 
\ and error. ("Transparency No, 7) Thifc 

figure is translated into -a more usable form 
in Table 1 on page 12. (Transparency 
No. 8) H , 

To show participants how to use this table ("it , 
sometimes helps to call this table "Give-or-Take" 
Table)/ use an example from the Wilson report; the 
fourth gjrade had a gain of 1.3 NCEs based on 28 ' * 
students which the table says will have an error of 
3.0 NCEs. Therefore, the teacher at Wilson could say 
that based on the results *of this year's evaluation/ 
the effectiveness of the .Wilson program" for ^fourth 
grade students is 1.3 NCEs "give or take" S.O^NCBs. 
The first thing that participants will nofre is that 
this means that the NCE £ain could be anywhere between 
-1.7 NCEs to 4.3 NCEs. In tarn, this means that next 
year if nothing is qhanged, there is a , change that the 
NCE gain qould b4 negative". Ideally, the confidence ^ 
band should not include zero. Additional note: Thi's 
c is also a good place to reinforce the distinction^ 
between "N" and "Membership" on th€^ reporting forms: 
Sometimes participants want to use "Membership"- 
instead of ^"N", when using* the, give-or-takfe table. 

J > ■ . V * 

Participants shQuld graph the confidence bands scr they 
get £ different picture. H^nd out blank' graphs for 
them to work on. WalK them through an e&mple on a * 
transparency. • " * \ 

r 

In districts that have small classes, the participants 
will notice that the error in an NCE gairris Very high 
and they will have to show substantial NCE gains to 
avoid having zero in the Confidence band. Page 15 of 
the' Interpretation Guide gives some suggestions for 
small clas.ses. - 1 y 

m Combine NCE gains acrass buildings to look at* 
district NCE'gains, 9 - 

4 

• Combine NCE gains from adjacent grades if program 
• is essentially the same, or •* * 

• Combine NCE gains across years. 
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Ma terials 



Strategies 



Comparative [Enterpretations — Simulation 

|o help Wilsjon decide whether the NCE gai,ns .should be investigated it^raay 
Wfc^p to loo )<* at 1 some comparative information. Handout materials that 
contain comparative information: 



Handout 
•15.272, 
15.273/ 
15.274 

Transparency 
No. 12 

4 

"Transparency 
Flo. ]# 

Handout I-H-5 



Transparency 
No. 14 

Handout" I-H-6 



Cherry Hill Elementary 's report. ^ N „ 

Wilson Elementary 1 s report from last year* 
State average NC R gains «_ . 

District objectives \A " 

How do Wilson's results compare to Cherry Hill Is? 



\ 

/ 

They are higher* Warn participants that when comparing' 
gains, they al^o have to remember -confidence bands. 
Hand out graphs that have both*Wilsdh* and Cherry Hill 
grade four*ori them and graph confidence bands based, on 
the give-or-take £able. (Transparency No. 8) * ; If the 
twb bands, do not overlap, then participants can be 
more sure that there really is a differenced , Use an 
example from the Wilson data: The actual raean^galn 
for third graders in a project like this cQuld be as 
lo^Hfes -1.0 while in the fourth grade it could be as 
high as 4.4. . : Therefor^,, ^e're not sure whether tnird 
grade gains are actually lJ.gher or lower thai} fourth 
grade gains. JP 

Mention that ^comparisons should take into consideration 
«%e possible differences in student types, 
administrative, community, and>parent support, or 
other program factors as well as looking foj fa g 
comparability as to? the tjfests used and t;he eflraiuation^ 
cycles. * 



Transparency 
Nos. 15, 16 . 
Handout lrH-7 



• How do Wilson 1 s-.results -compare to last year? 

Eairly consistent. We could rule out any 
circums'tances peculiar ta this school year. 

/• . • * 

• How do' results compare to 'the State average gains? 



^Transparency 
No. 17 . 



They 1 re lower. The* same idea of > confidence bands 
hpl<}s. The State's, gaind are "based on such a large 
population, however, that the «band will be much 
smaller. You can graph thik also for coniparison 
purposes. 
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Materials 



Transparency 
No. 18 



o * 



How do 
objectives? 



7 — : 

Wilson 1 s ' g 



if - 

gains compare * to *the district 



Strategies 



They ar,e lower, but note that Centennial School 
District has set fairly hi^h, gokls that could be, 
unrealistic. Previous NCE gaips were probably not 
referenced when/stating this objective. / 



Exploratory/Diagnostic Interpretations — Simulation 



Transparency 
Nb. 19 

•Handout 15.279, 
15.2^0 * J 

o * 




The last section of the - Interpretation Guide suggests • 
some areas that may be investigated in a project to ' 
identify the strengths and weaknesses. 'This section 
is only touched on briefly and can be more thoroughly 
coveted in workshops or needs assessment or 
implementation evaluation. 

As a lead in to the discussion, finish the simulation 
by handing out the final two pieces-: The Director's 
Summary and the Evaluatojr's Summatry. . Ask the 
participants* to work in small groups to identify what 
they feel may be causes of the results from the 
evaluation report. Three have been built into the 
simulation: 

1) The objectives of^the Cherry, Hiir project 
emphasize comprehension more than Wilson's. The 
test used for evaluation is the reading- 
comprehensive subtest: " (Note that an assumption 
that u$e made earlier was. that the test matched, 
the 'curriculum.* This is a point where the 
presenter may want to •emphasize that tests should 
be re-examined for content validity on a regular 
basisl) 

2) The Titie I teacher spends more time or 
administrative task^than with actual teaching. 



•3) 



Most of £he students 'already had achieved the 
objectives in the Wilson program when they 
entered. 



After the groups haJe' worked for about 15-20 minutes, 
"ask them. if they've discovered some possible 
problems. Discuss these in the large group. 

°Finish the workshop by discussing what comprehensive 
, evaluation involves. See , page 33 of the guide for a 

br|ef discission.- Explain that this workshop is not 
-designed to go into this area, but the participants 
, are not to interpret this as meaning that it's not 

important. < 

9 " 1659D 
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Section^ li: overhead Transparencies 
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* * 

• . Used. Sensitive Test 

✓ 

• Followed Publisher's Instructions 

• • Controlled Testing Conditions 

• . Followed Evaluation Model Guidelines 

• Met Tachnica! Standards- 
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Wilson Elementary School 
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TITLE I READING PROJECT DESCRIPTION 
Wilson Elementary School 



Background of Projec 



The Titla- I reading project in Wilson Elementary School focuses on the 
development of reading skills in 3rd, 4th and 5th graders. This project has 
been operating for three ye^rs, using the 'same basic structure. 

Instructional Activities and Services 

A reading resource lab is staffed by a Tittle I teacher and one full-time aide, 
both working .8 FTE. Students are scheduled int'o the resource ropm for 
approximately one-half hour per day, where they^ work on projects in small 
groups. Instruction is based on an individual analysis of each student's 
needs, although the instruction may involve groupings of objectives and 
students may work on several "objectives at the same time. Mastery of specific 
objectives is determined by a teacher-developed system of measuring 
performance. In general, ten students work in the room and interact with the 
teacher on £n individual basis. 

The Title I teacher coordinates the activities of the resource lab. The 
teacher's responsibilities include supervision of the aide; consulting with 
classroom teachers regarding special needs of target students; individual 
diagnosis; design of instructional strategies to meet needs of target 
students; di'rect instruction to target students ip the" resource lab; and 
assessment- of student progress on objectives. 

The aide assists the teacher in any capacity as determined by the teacher. 
Evaluation 

The Wrlson Elementary School based its most recent evaluation on a 
norm-referenced evaluation model. This evaluation model compares the average 
score of Title I students to national nofms at two points in time. A test 
administered prior to the start of the program is used to set the expected 
percentile standing of the Title I students on the posttest. The expected 
percentile is the average percentile standing of the Title I students at the 
pretest. If there were no Title I program, the average posttest percentile is 
expected to equal this value. The difference between the observed posttest 
standing and the expected posttest standing for the grouo is the" measure of 
Title I progxam effect. For the reading program, the reading subtest, Form Q, 
of Level IV (3rd and 4th grade) afad Level V (5th grade) of the Comprehensive 1 
Achievement Battery. was given and the total reading subtest score was used in 
the evaluation. The test was administered in October and April. 
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School Code. 



District Code. 



Project Identification Title I Reading Project 
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ACHIEVEMENT INFORMATION 
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Th^s Year s Achievement Impact Report* 
Cherry H f ill Elementary School " ^ 

Last Year's Achievement Impact .Report 
Wilson Elementary^ School 

*This Y.ear's State Title I Achievement Impact Report 

Program Objectives for Centennial School District, 
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NCE Gain by Grade, Reading Projects 
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NCE Gain 
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Program Objectives for Centennial Title I Reading Project 

As a result of program participation'/ target students will demonstrate 
improved reading and language skills: <» 

1. Target students will show an increase in overall reading skill 
as demonstrated by an increase of two reading book levels during 
the project year, 

2. Target Students will show improved reading comprehension skills 
as demonstrated by an average gain of eight standard scores 
(NCEs) above their expected score without Title I. The reading 
comprehension subtest of the Comprehensive Achievement Battery 

4 will be used to measure reading skills. 

* 

Individual students in the target population will reaeive diagnosis and 
prescriptive instruction focused on each child's needs* 
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C. Graphs for Showing Wilson's Results 
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TITU3 I DIRECTOR'S ACTUAL PROJECT SUMMARY 
Wilson Elementary jgchoo£/Titl$ I Reading Project 



The Title I reading project this year has been moving fairly well. I fc 
feel that students made a great deal of progress, although the program 
evaluation indicated that we did not meet all of our goals, 

* 

Overal^Program 

Scheduling students into the resource room for half an hour a day seems , 
to work jwell. As always, we noted a good deal of absenteeisn on Mondays, 
but attendance was high Che rest of the week, k 

Several of the new classroom teachers made little use of the resource 
room and our special program* Several of them seemed to prefer dealing 
with Title I students in their own classrupras. But we are working on 
better procedures for identifying all Title I-eligible students next year, 

In my dual role as project director and teacher, I have been able to 
observe the program both from* the instructional and f ran the 
administrative point of View, and I feel very good about our progress. 
Thevaide an3 I worked well together and, as a team,' were able to keep 
students interest^ and involved in^ihe" program, ,As part of ray record- 
keeping, I kept track of how we both spent our time over the past year, 
A table summarizing our activities is included. I felt that we ^pent 
somewhat longer than necessary in recording student achievement and 6n 
providing direction (rather than instructional help) to students. 
Several recommendations are included to avoid these problems in the 
future, ' ~ % 

' • ;,. ' 1 i !_ 



Tasks 



Percentage or" 


■^ime on Tasks 


Teacher 


Aide 


11 


2 


4 


8 


22 


* 26 " 


'13 


4 


12 


19 


13 
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8 


21 


7 


7 


8 


8 


2 . 


0 



Administration 

iesting students*— formal tests * 
Checking mastery with informal tests 
Developing new proj^pts o$ materials 
Correcting! assignments 
Giving assignments 
Monitoring stqdept work on projects 
Talking with students m a 

Talking with each other 
Talking wijth parents 
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Student Mastery of Objectives 

While our project's performance in terms of NCE gains is somewhat 
disappointing/ we have made observable* progress with the students. 
During* the past year, as students mastered specific objectives, we marked 
the objectives on a student achievement record. At the end of each 
month, my aide records the number of Title I students mastering each 
"objective. These monthly charts were^used to examine student progress on 
objectives. 

The program objectives cover four different skill areas: phonetic 
analysis, structured analysis, vocabulary and reading comprehension. 
About half of the students entering the program had already mastered at 
least 73 percent of the phonetic analysis objectives, 58 percent of the 
structural analysis objectives, 36 percent of the vocabulary objectives 
and 17 percent of the reading comprehension objectives. By tfie end of 
the year, 80 percent of the students had mastered all of the phonetics H 
and vocabulary objectives, 84 percent the structural analysis ones and 44 
percent the comprehension ones. These figures seem to indicate that 
progress is made on all of the objeptives the program. 

Recommendat i ons v * 4 

Before*, school begins in the fall I would like to be able to spend some 
time preparing individualized packets for each student who will be in the 
resource rocra. If each student received a packet detailing instructions 
for several alternative activities for .each objective, explaining and 
encouraging the use of different learning options would be greatly 
facilitated. 

We. also need to spend some time with our recordkeeping systems, so that 
when student- teacher and parent- teacher conferences are held it will be 
easier to access the relevant information. I recommend sending my aide 
and myself to the recordkeeping clinic sponsored by fcbe publishers of the 
Special Readers Book Series we .are using. 
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I Evdluator's Summit gy 



Xbis report "summarizes a^. evaluation of the Wilson Elementary Reading 
Program over the past year. • f 

fc Impact Evaluation , ^5iven the size of Centennial School District and ttfe 

- lack qf an adequate comparison group of students at Wilson, Model ,A was 

^ used to, evaluate the Title I program. In conducting the study, Model A 

• ' guidelines were followed. . Students were tested at. the fall and spring 

empirical norm dates of the* Comprehensive Achievement Battery, student 
i / selectioh was independent of pretest scores, all* students Vere tested 

p , within a ,short timespan, and the test administrators followed the 

publisher's directions. fc 

^ The impact data showed October to April gaifts of 2,1'NCES for the third 

^ graders, 1.3 NCEs for fourth graders and 1.7 NCEs'for fifth graders. 

These gains mean that as a group, third graders moved from the 17.0 
percentile to the 21.0 percentile, fourth graders from the 18.1 to the 
* 19.8 percentile, and fifth grader^ from the 13.8 to the 15.7 percentile. 

The impafct data are based upon the total reading subtest of the 

# 1 4 Comprehensive Achievement Battery (CAB) . 

It should be stated that any NCE gains greater thao 0 indicate some 
program effect. A comparison of Wilson gains with the stated program 
objectives shows a much siiialler gain than expected, however. . 

0 Even a comparison %p the gains achieved by students at ph^rry Hill 

Elementary School, a similar school with a similar program, -shows that 
the Wilson gains., are probably not as large as desired. A comparison of 
th e reading programs at Cherry Hill arid Wilson may indicate some possible 

* -\ explanations for the differences in achievement. For example, the 

' proportion of objectives in each skill area differs dramatically for the 
- , , tw6 schools. The areas of objectives and number of objectives in each 
^ * area are listed below. 

/ Table 'I-' s ' * 
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No. 


of Objectives « < I 


1 Area 


Wilson 


Cherry Hill 


* " 

Phonetic analysis 


22 


22 " 


Structural analysis 


12 


12- 


Vocabulary 


11 


11 


Reading comprehension, 


< 6 . < 


21 



Given the differences in 1 tile level of objective for the two- programs, 
April test scores for* Title I students in both schools *were computed for 
all reading 'subtests of the CAB. A description of the J:est content is 
a^tacbey 

\ 
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A further, analysis of the Wilson program was based upon student mastery 
of objectives . Overall student mastery is summarized in the Title I 
director's report ♦ Further study of student progress rates also showed 
that roost student mastery of phonic analysis and structural analysis 
objectives occurred in the first month of the program. Little progress 
was seen during .febe last half of the project. This would seem to 
indicate some problem in the program structure. 
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Table 2 

Percentile Ranks <of Title I Studenta on the 
Coriprehensive Achievement Battery 
(April Testing) 



Elementary 


Subtests 


Total 




Grpde 


School 


Phonetics 


Word Analysis 


Vocabulary 


Comprehension 


%ile 


NCE 


3rd 

t 


Cherry Hill 
Wilson 


38 
37 


35 
36 


25 
22 


26 
14 


29 
20 


38.3 
32.0 


4 th 


Cherry Hill 
Wilson 


39 5 
45 { 


i 


42 
38 


31 • 
25 


. 35 
15 


37 
20 


42.8 
32.1 


5th 


Cheery Hill ^ 
flilson 


37 




30 
30 


28 
" 22 


22 
10* 


25 
16 


35.7 
28.8 
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The Comprehensive Achievement Battery E*ovided a content description of each 
subtest. You copied the description for the four reading subtests which made up the 
Total Reading score used to evaluate the program. For each general content area, 
the number of test items measuring the skill was listed. 

» % . 

Number of Items 
Level IV Level V 
Skill Area (3rd&4th grade) (5th grade 

Phonetics Skills Subtest 



1. Matching Oral Sounds : Find words containing the 10 
same sound in words read aloud by the teacher - 

2. Matching sounds in printed words : Find words containing 
the same sounds as words printed in the test manual. 



10 10 



Wor<3, Analysis Skills Subtest 

1. Verbs : Select the appropriate verb endings. w 7 7 

2. Adjectives : Select the appropriate adjective endings. 7 7 

3. Ccfcfcinations.; Make contractions. and confound words. 6 . 6 



Vocabulary Skills Subtest 

1. Single Meaning : Select the correct definition of a 15 10 
word .that has only one meaning. 

2. Multiple Meaning : Select the correct definition of a 15 , 10 
word that has more than one meaning. 



Reading Comprehension Skills Subtest 

1. Restate Material : Recognize a restatement of material 10 10 
that is explicity stated in the passage; select a 

specific detail that is stated in the passage. 

2. Sequence and ySuromar ize : Select the proper sequence of 10 10 
ideas or actidn in the passage; select main ideas or 

choose an appropriate title for the passage. 

3. Draw Inferences : Rebognize material that is implied 10 10 
' but not specifically stated in the passage; recognize a 

characters motivations and emotions; select probable 
reasons f or* actions. 

4. Apply to^Hew Situations : 'Recognize a valid example of 
something* stated in the passage;* choose a likely 
outcome with one variable changed; , select correct 
applications of the information in the passage to a 
new situation. 

5. Logical Relationships : Recognize the relation between — 10 
premises or sections of a passage; recognize validity 

of procedures and variables in science and social 
studies passages; distinguish between fact and opinion; 
choose statements or examples of tech'niqtae and 
point of view. * 
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INTRODUCTION 

s 

So you^ve completed the end-of-the-year report for your Title I p'rograra. 
You've described your program, *averaged pretest* and posttest scores, and 
calculated NCE gains. There 1 s a feeling of relief yet also one of 
curiosity. Surely there must be more to evaluation than completing 
forms* What do these results tell you? Did your program work? Are 
these numbers in front of you acceptable? Are they low? -High? Can you 
use thera to tell you anything about your program? Are there parts of the 
program that should be revised? 



What this guide will flo 

This guide £s designed to take you from simple explanations of what an 
NCE gain means co fcore complicated concerns such as. where to look in the~ 
program for the strengths and weaknesses* To do this, the 'guide is 
divided into three major sections. 

\Section One deals with interpretations of NCEs and NCE gains. You will 
find explanations as well as alternative ways of describing gains. Also, 
problems of interpreting gains for small class sizes will be discussed. 

» 

Section Two builds on th&se initial interpretations by comparing^ these 
gains to different standards. These standard^ include last year's 
results, other projects 1 results, state averages, or objectives set by 
your program. 

• 

Section Three discusses aspects of your program that could be causing the 
project outcomes you have observed. You can use thi* information to 
diagnose program strengths and weaknesses. 



What this guide won't do 

Each section^ this guide progressively adds to the interpretation of 
your results. Sirfee Titly I programs differ widely in their 
'characteristics, providing you with absolute results would e be 
impossible. The final interpretation can only be made by you. -This 
guide will not, therefore, provide you with the answers to all your 
specific project questions. Hopefully, it will provide you with the 
necessary background information to discover those answers on your own. 

* 

What you need before using this guide 

To use thi-s guide, you must be able 'to assume that your data are a valid 
reflection of your program's impact. This means that you have: , j 

• Used a test which is sensitive* enough to detect the effects of 
the project 



# Administered the test according to publishers instructions 
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• Controlled for any conditions that could have invalidated the 
test results (such as room conditions, anxiety, extraneous 
noise , etc. ) 

• Followed guidelines for particular evaluation models (such as 
testing near norm dates, functional-level testing, etc. ) 

• Checked the»data according to the Title I Evaluation Technical 
Standards--representativeness, reliable and valid instruments 
and prt>cedures, minimization of error, and valid assessment. 

It is next to impossible to have perfectly valid data to interpret. 
Errors will creep in somewhere. However, to the^ extent that you have 
followed these guidelines, you can be more confident in using these data 
for decisions about your program. 

Where to begin in the guide 

As mentioned earlier, the guide is divided into three sections, each raQre 
advanced tftan the one before it. You may 'not have to start at the 
beginning and may want to begyi work somewhere later in the guide. You 
will probably find it worthwhile -to at least leaf through some earlier 
sections to assure' yourself of that familiarity. 

A running example 

To illustrate the points made throughout the manual,, a running example is 
provided based on the experiences of Centennial School District in 
interpreting their Title J results. Each part of the example is 
highlighted with a box. ' 



Example — Project Description 

During the 1980-81 school year, Centennial School District 
implemented Title I programs in .elementary reading and middle 
schoo^lan^uage arts. A -Title I coordinator was assigned to 
monitor the instruction and to conduct the evaluation for these 
programs. 

The reading program was implemented in third, fourth, and fifth 
grades in .both Wilson Elementary and Cherry Hill Elementary." 
Students worked in a resource room staffed by a teacher and an 
aide for about, 30. minutes each day. * 

r 

* Ttoe language arts program at Centennial Junior High included 
seventh and eighth* grade students. Remedial instruction was * 
provided by the classroom teacher while the rest of the class 
worked on other assignments. 
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SECTION ONE 



WHAT ARE NCE GAINS? 



What are NCEs ? 

NCEs (Normal Curve Equivalents) are the scores used to report Title I 
program evaluation. While there are already many types of test scores 
available, NCEs were developed to avoid problems which occur when using 
other scores for program evaluation. Let's compare NCE scores to 
percentile ranks with which you are familiar, 

A percentile rank compares a student's performance on a test with the 
performance of students in the norming sample for that test, yielding the 
percentage of norm group students who fell iSelow a particular score. An 
NCE score, also represents a student's level of achievement compared with 
students in the norm group, but using different units, 
* 

To evaluate ,the effectiveness of your program you need to average the 
scores of your Title I students to determine the performance of the^-whole 
group. To do this, you must use^a score that has equal intervals. You 
should average NCEs rather than percentiles, , 

The NCE scale and the percentile scale are matched at the 1st, 50th, and 
99th points. Thi3 can be seen in the figure below which presents a< 
comparison of the two scales. 



scaoL cum equivalents 

,,| mm,", m i ^j l ni i lYHfti i ji UMiiilP'hM ' MMH - ! 

30 40 -^50 " 60 70 ] 80 * 90 99 



10 20 
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I I ■■■ l l' TTTrtr i , i i ■! in . 



10 30 SO 70 0L 90 99 



Notice that the distances between ibints on the percentile scale are not 
equal. At the upper agd lower end? of the scale, the points are farther 
apart than at the middle of £lfe scale* 6n the NCE scale, the distance 
between o£ints is the same. For this reason* the NCE scale is referred to 
as an equal interval scale, $, 7-—^ 

You can also compute the "average using expanded xstandard scores since 
these scores have equal intervals, too. (Note that the publisher of your 
test may use a different name for these scores—standard scorejgfccpanded 
scale score, growth sca>e value,) These -scores roust generairflBT used if 
you have tested your students at a level -below that reeommenderoy the 
test publisher. 
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The advantage of NCEs over expanded scores is that NCEs me^ the same 
thing regardless of the test used or the grade level, * Expanded scores on 
the other hand, are not comparable from one test to another or from one 
'grade .to another. 

In summary, NCEs have two primary advantages for program evaluation over 
other 'types of scores — they can be* averaged and they are comparable tt 
acros's, grade levels and across tests. 



What is the meaning of NCE gains ? 




:t^t 



Title I .students are pretested early in the ofograra and postt<fsted late 
in the program to determine how much. they have grown in achievement. 
This growth is due to regular classroom i-srStxuction, maturation, and home 
influences as well as the supplementary Title I instruction. To isolate • 
how much- of this growth is due just to your program, the Title I 
evaluation compares the performance Title I students with similar . 
students who did not receive Title I lh^ruction. Consider how this is 
done using Model A, the norm-referenced evaluation model. 

Mod%). A assumes that the norms for a test reflect the growtji expected 
with regular instruction. Students grow in achievement at different 
rates, the better student^ learning faster than the more disadvantaged 
students. On the average though, a student tends to remain at the same 
rank relative* to his peers. Thus, a group of students, whose pretest was 
at the 2dth percentile would b£ expected to have a posttest average at 
the 20 tn percentile. 



< 




20th percentile 



Posttest 
Spring 



Title I students, ' however , have hopefully learned more than is expected 
for similar students in the regular 'classroom. Any positive difference 
found between the average posttest and the average pretest for the Titi#^ 
I group is the gain that can be, attributed to Title I instruction. This 
gain is an indication that extra growth has occurred oyer and above the 
regular growth which woul^ have^been expected with only regular classroom 
instruction. 
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25th percentile r—j 
20th percentile ~ 



extra growth from Jtfle I instruction 



Pretest 
Fall 



Posttest 
Spring 



Since NCEs are used instead of percentiles to determine the impact of 
Title I instruction, the NCE gain for a #poup 6f Title I students is the 
difference between the average NCE on th£ posttest and the average NCE on 
the pretest. o H 



NCE GAIN 



NCE POSTTEST 



NCE PRETEST 



\ 
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The size of NCE gain you should expect may vary depending^on the^ grade 
level of your students, the, subject matter area served, and the nature of 
your program. It may be said, however, that moderate gains are 5-10 
SCEs. Gains of more than 20 NCEs are unusually high and may reflect 
errors in the way the evaluation was implemented. * 

* ~# 
What about a gain of zero NCEs? A common misconception concerning a zero 
gain* is that it indicates "no growth at a24" has occurred for the'group 
of TitlS I students^* This'is incorrect. With -a zer* NCE gain, the 
achievement level of your Title I group has increased from pretest to 

achievement level would have been expected 



•posttest, but this increase in 
with just regular classroom instruction. 




A negative gain indicatea thaTyour students did not grow as much as low 
achieving students in other schools. ^ Alternatively i< mijght suggest ^that^ 
the testing or evaluation were n£t done properly*. 



A 



Ex ample-'- NCE Gains * % *'* % ' 

To evaluate Centennial's Elementary Title I program, a reading 
test ttfat seemed to match the curriculum was administered fall 
and spring, the following results were obtained for "Wilson 



Elementary: 








***** 






Grade 


N 


Pretest 


Posttest 


NCE Gain * 






• * 

3 

5 


27 
28 

- 22 


29.9 
- 30.8 
27.1 


32.0 

32.1. 

28.8 


2.1 






While tlfcre 


were 


positive' gains 


at all grades.. 


the gairfs were 






small. 
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How can project' gains oe more easily explained to others ? 

: V 

When reporting to local audiences such^as parents or administrators, you 
can report the results of your evaluation in ways which are more familiar 
to these gtroups. NCEs are useful ftfr computing the group average and for 
aggregating across buildings or.graaes but are probably unfamiliar to 
these audiences. Some alternative methods include: converting NCE 
averages to more familiar type s< of scores, graphing NCE gains t<? 
emphasize trends, and describing the objectives achieved by the students^ 
irf the program.- 

CQnverting NCE averages . Once you have computed the group's pretest and 
posttest average NCEs, the average can be .converted to other scores. For 
example, you can express the extra growth of your students due to Title la 
as an improvement in their percentile rank. While percentiles should not 
be averaged, they can be used for local reporting. 

To express your gain as* a change in percentile rank, simply convert the 
x pretest and posttest average NCEs to percentile^ working backwards V* the 
percentile to the NCE conversion table provided below. ■ 



Percentile to NCE Conversion Table 



% 


NCE 


% 


NCE* 


% 


NCE 


% 


NCE 


% 


NCE 


1 


1.0 


11 


24.2 


V 1 


33.0 


31 


39ft 


41 


45^ 


2 


.6,7 


12 


25.3 


122 


33.7 


32 


40.1 


42 


45.8 


3 


10.4 


• 13 


26.3 


(23 


34.4 


■ s . 33 


40.7 


43 


46.3 


, 4 


13 . i s 


14 


27.2 


24 


35.1 


34. 


41.3 


44 


46.8 


5 


15.4 


15 


28.2 ^ 


25 


35.8 


35 


41.9 


45 


47.4 


6 


17. a. 


16 


29.1 


26 


36.5 


36 


42.5 


46 


47.9 


7 


18.9 


17 


29. *9 


27 


37.1 


37 


43.0 


47 


48.4 


3 


20.4 


18 


30.7 


28* 


37.7 


38 


43.6 , 


'48 


48.9 


9 




19 


31.5 


29 


38.3 


39 


44.1 * 


49 


49.5 


10 


23.0 


20 


32.3 


30 


39.0 


40 


44.7 


50 


50.fr 



/• 



If tjie pretest and posttest NCEs were 33 and 39, the corresponding 
percentiles would be 21 and 30. Thus at posttest timer the average 
Title I student in your program did bet^teif th^n 30 percent of the 
students in the national norm group, and 9 percentile points better thar 
at the pretest. ^\ , 

Graphing gains . A good graph c^m draw the refers attention to trends* 
and basic concepts even if the^'dp not fully undeTstSmd the underlying 
scale. A graph of NCE gains might focus the readers 1 -attention on the * 
additional growth due to'Title'L. A graph of tfce average pretest and 
posttest expanded standard sccfres could b^j^^l to- compare expected 
growth (group maintains* same percentile) with actual growth. The 
following graphs show gains graphed using NCEs and expanded standard 
scores. 



9 

ERLC 



1/81 




r 



15i 



A4dib«U( AcfcitvvwHt 
Out tt Titlt I 



TitK 



4 5 
GRADE 
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350 
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Q 



260 
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Showing content mastery father than talking in generalities about 
achievement or growth, it may be helpfUl'to focus'on the specific skills 
or objectives learned. 'If you have good ^records pn the objectives 
mastered by students, you can talk about the percent of students 
mastering a set of objectives or the percent of . objectives mastered* If 
you have used the_test publisher's scaring servicer£7~ybu probably have- an 
item analysis k eyed "ob] ec t i vgs that could be used. When hand scoring, 
you could create your own item analysi's by' determining the number of 
items e*ach student <yot right for clusters of iteihs measuring the same 
oojective. * / 



Are project g^ains accurate ? * . 

While test, scores are generally a good indicator of % a student's 
^achievement, it is not possible to obtain a perfect measure of 
achievement. As you know, there are' many sources of measurement error 
which cart affect the test score a student obtains ♦ These include 
^fluctuations in the student's mood or motivation, momentary distractions 
during the test, guessing, and familiarity with the item/ format ♦ Because 
o$ these factors, the student's obtained score at onfe yime will 
differ — sometimes higher, sometimes lower — from the test score obtamed 
at another time. .That is, the*score tells" you roughJ,y what the* student 's 
true achievement level is, but not exactly. ^ * 



When project gains are computed, the observed gain is affected by errors 
from both the pretest and posttest. By averaging across all the students 
in the Title I class, how*eve<r, this error is reduced since the various 
sources of error tend to cancel'each other. The gains based on only a 
few students can vary greatly depending or* the particular sample of 
students included. G^ins based on many students will be much more 
accurate. 

Since you generally want to know how effective your program is with 
Title I students in general — not just with this year's gro'up— *there is 
another type of error thai: affects your result^ the luck o.f the 
draw, one year you might have a good group that would make your program 
looK t&tter than it should or a bad group that would make it look worse. 
This is particularly a problem if the grou^is smail since the scores of 
just one student can affect the results for the. whole group. . * 

i 
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Consider the following example of posttest scores for a Title I class 
with four students. 

* 

23 
37 
16 

21 

Total = 100 Average score » 25 NC£s 

Now suppose that the first student was ill the morning of the test *(fi^st 
type of e'vor mentioned above) ->an£ that the second student never tri^T to. 
learn during the program (second type of error mentioned above). E^fth 
students 4id less well than they should nave on the test. In thier case 
the scores might have been: 

13' 
27 
16. 
24 
80 



Total 



Average score = 20 NCEs 



In a class ofr four students, these two errors, had a large effect on the 
posttest average for the group. In a class of 25 students, the effiect 

would not have k^een very big, 
i 

You can estimate how error affects the accuracy of your results by 
referring to the following graph (see Technical Note 1): 




» is » a 

dass sire 



As ^you can' see r the accuracy of an NCE gain vafi^s depending on the 
number of students in a Title I class.- The amotint of error decreases as 
class size increases. Yojj can also see that 'the size of the error begins 
to become more constant When the class size is 15 or mor#* The error 
sharply increases- as the class size becomes smallePthan 15. Note that 
-^ojUgs 3ize here refers onlv to students included in computing the ^ainS/ 
that is, students for whom there were both pretest and posttest acores. 
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How do you krow where your true gain lies ? 

I 

While you don't know what the true gain is exactly, you can. estima.te a 
band within which you can be quite confident the tru^e gain would fall. 
To Construct this band, you can use the table below. The confidence band 
is your project 1 s .gaift, "give Or take" the amount o^ error given. for your 
class size. The table is based .on the pj^ceding graph. 




f "Give or Take" Table 



Class 3ize 


Error 


(N) 


(NCEs) 


. 2 


16.0 


1 3 


11.3 


4 


9.2 


5 


8.0 


6 








8 


6.1 


9 


5.7 


- 10 , 


5.3 


11 


5.1. 


12 


4.8 


13 


4.6 


14 


4.4 


*9l5 


4.3 


16 


4.1 . 


17 


4.0* 


13 


3.9 


19 


' 3.8 


20 


' 3.7 



Class Size 


Error 


(N) 


(NCEs) 


21 


3 . o 


\ 22 


3. 5 


23 


3.4 


24-25 


3.3 


26 


• 3.2 


27-28 


3.1 


29-30 


3.0, 


31-32 


2,9 - 


33-34 


2.8 ' 


35-37 


2.7 


38-40 4 


2.6 


41-43 


2.5 


44-47 


2.4 


43-50 


2.3 






75 f 


1.9 


* 10t) • 


1.6 


150 


1.6 


• 200 


1.1 



Suppose that the gain for your Title I group of students is J NCEs and 
you fcave scores for 17 students. < Using the table, you would find that 
for a class of 17 students, the error for «thisflain is about 4 NCEs. . 
In this case, the confidence band is formed by Subtracting . the error, 
4 NCEs, from the NCE gain of 7 to get the lower boundary of the band.- 
The upper boundary of the confidence oand is found oy adding the error to 
the gain. 
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t 




% 


r 

V 


« 

• * 


\ 


•4 

Lower Boundary, . 


* a 


• ' U 

Upper Boundary 


NCE Gain-^Error 




NCE G^n + Error 


•4 r 



7-4 = 3 • 



7+4 = 11 



You can be quite confident that tfie tjrue gain. of the program wa£* 
somewhere between 3 to 11 NCEs. The* program" did have a positive 'effect. 
The confidence bancf can be pictured graphically as: 



P 
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If the observed gain, had been 3 NCEs, however, the confidence band would 
have been -1 .to 7 NCEs, In that case, you could not be sure whether the- 
trUe gain was greater than zero. That is, with a gain of 3 NCEs with an 
N of 17 you cannot be sure whether or not your ptogram was effective in 
boosting the achievement of your students any more than would have been 
expected with just regular schooling. 



Example — Confidence Bands for NCE Gains 




< 

The Title I coordinator thought that the gains from Wilson 1 s 
reading program were rather siriall and wanted to see how accurate 
the results were. Using the "give or take" table, the 
coordinator made up the fallowing worksheet: 


Grade N 


Gain Error G - E G 


+ E 


3 27 

4 ,28 

5 22 


2.1 3.1 -1.0 
1.3 3.1 -1.8 
1.7 3.5 -1.8 


5.2* 
4.4 

"5.2 


When the confidence bands were graphed, the results 


looked like: 
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It should be quite apparent from the "give or take" table that when the 
class size is small*/ gains are not very accurate, ^he confidence band 
can be quite large with a class size less thanj.5. One solution is to 
increase your class size by combining the scdfys from several classes 
befdVe interpreting the project's effect. Theje are several possible 
ways *to combine scores. 

► 

Combining Schools . One possibility is to combine the scores from the 
same grade across schoofl. If Title I instruction is given to fifth - 
graders in Schools A, B and C, then it would be possible to combine the 
fifth grade students 1 scores from these three school buildings. As the 
figure below shows, this greatly improves accuracy. 



GAINS 



*id* A 



Vdq. t Bid* C 




V 
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Containing grades . l£ your Title I project is similar .across grades, 
another alternative is to combine the scores from adjacent grades to 
increase the class size. For example, suppose your Title I project 
includes the third, fourth, and fifth grades but in the third grade only 
three students are participating in Title I. Instead of interpreting tjie 
NCE gain for only the third grade students, V°u could combine the 
students' scores from the third, fourth, and fifth grades. However, the 
NCE gains for elementary students are generally larger than the gains for 
students in the junior and senior high. schools; so you may not wantf to 
combine secondary results with elementary project results. 

Combining years . A third possibility is to combine the data across years 
of the program. If the same teachers, materials, tests, and such have 
.been used in the Title I .program f or . two (or more) years, you can-comoine 
the scored for students in each grade over" .both years. For example, a 
program with scores for 10 fourth graders in year 1 and 8 fourth graders 
in year 2 could combine the two sets of scores for a total N of 18, 
reducing* the confidence band by about a third. ^ 

When determining the NCE gain for combined scores, do not average the NCE 
gam from one class with the NCE gain from another class by adding tnera 
and then dividing by 2., This kind of averaging will produce an Incorrect 
gain since the origmal'gains were based on different class sizes. 

I There is a short cut method for combining scores if you have already 
x^computed the gain for each class you want to combine. Simply use the 
following Worksheet to multiply each gain by the number of students (N) 
used to compute it. Divide the sum by -the total N to get the overall 
gain. 



Year School Grade N Gain N x Gam 

* ■ * 

% 

Total: Tocal: 



Overall gam = Total N :< Gam _ 
» Tocal N 
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Example—Combining NCE Gains for Interpretation 

Tht language arts program in Grades 7 and 8 at Centennial Junior 
High has only a few students at each grade level. The Title I 
coordinator decided that the combined results of the two grades^ 
would be more meaningful than for either grade separately. The 
coordinator also decided to lump the data from last year and 
this year together §ince the program had been implemented in the 
same way and the. same test was used both years. The results 
were as follows: V 



•Year 


School 


Grade 


N 


Gain 


N x Gain 


79-80 


' Jr Hi 


7 


2 9 


12.1 9 


24.2 


79-8,0 


Jr Hi 


8 . 


5 


6.4 


32.0 ■ 


80-81 


Jr Hi 


7 


4 


4.7 


18,8 


80-81 


Jr Hi 


• 8 


'3 


-0,5 


- 1,5 


Overall gain = 


73. t = 


14 

5.3 NCEs 




73,5 



14 

• - « 

Confidence band = 5*3 & 4.4 NCEs 

r 

The overall gain for the language arts program (based on 14 
students) was much more encouraging tnan any of the results 
based on fewer students. Little conf ideiKre can be-placed, for 
exarapler'i"Tr"t3Te"i2'^C£ gain at grade 7«since the error, band for 
only two students is 16 NCEs. * 
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SECTION TWO 



x COMPARING PROJECT GAINS TO STANDARDS 



The focus of* the last section was on how to determine whether your 
Title I students learned more than they would have without Title ^ 
instruction. That is fine, but it does not go far enough. You still dd~ 
not know how good or how efflctive the ptoject was. 




\ 

You have probably encountered a similar problem in trying to attach 
meaning to a student's test score. Take, for example, a student, John, 
who scored 35 on a 5G-item math test. You do not know whether 35 is a 
good score or bad score without comparing it to some standard. You wpul 
want to know how John did on the test before taking the class, how other 
students in his class did, how other students at his ability -level did, 
how you think he should .have done. Using such information, you can 
construct standards for judging the worth of a score of 35. 

We might find, as in the example below, that while John did not do as. 
"well as the class average, he did improve substantially over his ^ 
performance before instruction and did exceed your expectations for h'im. 



50 
a. 40 
30 
20 
10 



o 

o 
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John * His Pretest Class Your 

1 Average Expectation 
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How do you know bow good- your gains are ? 



/ 



You cannot tell how effective a Title I program is by simply Rooking at 
the size of the NCE gain. You might not expect the same si^e of gains 
for your secondary program that you would for your elementary program. 
You might not expect the same gains for a~ limited, poorly* funded project 
that you would for one that involved intensive remedial instruction. To 
determine bow good or bad .the program was, you must have something to 
compare it to; you must have standards. . < 

The fallowing* table suggests possible sources of ^standards that you may 
find helpful and presents the questions based on a third grade project. 
Each comparison asks sbraething different aoout that example project. 



4 



Question 



'Possible Standards 



"How did ray Title I third 
grade program compare to 
third grade Title I programs 
across the state?" 

* * 
"How did ray third graders do 

corapared-to last year?" 



Average gains for Title I projects 
across state for the same grade 
and subject 



Performance of s^rae Title I 
Students last year 



"How did ray third grade program 
do compared to my th^rd grade 
program last year?" 



Gains from last year's project 
at eaph grade 



"How did the third* grade program 
do compared to the second and 
fourth grade Title I classes?" 



Project gains at adjacent grade 
leve Is 



"Did the third grade program 
make the project goal of five 
NCEs gain?" 



Project objectives based on 
professional judgment 



The process of making judgments about your program is, by nature, 
subjective. The Comparisons that you select will provide rough standards 
by which to judge your program, but the comparisons cannot be made 
withput qualifications: 

• Use several comparisons; do not rely on just one. 

• Temper ydur judgments oased on the similarities and differences 
between the chosen standards and the program being evaluated. 

In the example below, the results for a Title I project were compared to 
three standards; a) the project g^in from last year, b) the average gain 
across the state for that subject and grade, and c) the goal or objective 
stated in the project application. * The gain meets the project objective 
and is clos6 %p the gains observed for the average Title I program." 4 In 
addition, there appears to be improvement over last year's results. 
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GAINS 



How big a difference is important ? 



Your project's g&in will rarely be exactly ecpial to the standard. How 
big must the difference be between the ooserved gain and the standard * 
before it is worth considering? Is a difference of 2 or 3 NCEs 
important? Probably not. {tow about 4 to 5? You must consider ^he 
accuracy of both>gains before answering -that question. 

In the, first section tfte point was made that there is always some error 
involved i-n determining project gains. Constructing a confidence band* 
about a gain using the table given in Section One provides a way" of 
estimating the effect of error for each gain. If the confidence bands of 
two gains overlap; you cannot be confident that gains actually differ. 
Alternatively/ if the bands do not overlap you could conclude with some 
certainty that the gains do differ. 

In the example above, for instance, the Y€s31ts for a Title I project 
were compared to three standards. As shown in the figure below^a 
confidence band can be constructed around each gain based upon the number 
of students for each. Si*nce the state average was based dfc the scores of 
many students, the rand is very small. It did not make sense, however, 
to construct a band around the project objjectitfe since it was based^on 
professional judgment ra*her than computed from student test scores. 



GAINS 



>t t*ar t git* 




Tfci* T*w Utt Tut $l«tt *9|tCt 



You would conclude from this example that the project was effective in 
promoting tne achievement of educationally disadvantaged students. The 
project objectives were met and tfce results are similar to those observed 
for other Title I projects across, the sjtate in that: subject and for that 
grade. Thete was, however, a substantial improvement over the results 
from last year. It would be interesting to explore differences in the 
way the program was implemented each year to try to explain this finding. 
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It should be apparent that when gains are computed on fewer than 15. 
students, the confidence bands are'guite large and only very large* ^ v 
differences between gains can be detected. Project gains based on only' a 
few" students can fluctuate greatly. As discussed in SectionOne, the 
gains for several grades or^several buildings cah De combined to reduce 
the size of the confidence bands. 



In addition to looking for large differences between project gains and 
standards, we can watch for consistency in th£ results. A large gain 
relative to what was expected seems more reasonable if similar effects 
were observed at several grade levels or across several 7ears, 



Example— 

The Ti^le I coordinator decided that while Wilson's gains in * 
reading seemed discouraging, perhaps it was not unreasonable to . 
expect small gains for this type of program. Several gains were 
selected f^r comparison; Wilson's results from last year. 
Cherry Hill's results from this year, and the state average. To 
avoid the temptation to make too mucn qf small differences in 
gains, the coordinator graphed the confidence bands around each 
gain. 

* * 

Grade 4 Reading 





N 


Gain 


Error 


Wilson 1980 


28 


«tfl.3 


3,1 


Wilson 1979 


25 


2.6 


3.3 


Cherry Hill 


32 


8.4 




Stafe Average 


893 , 


6.4 


.5 



Th$. graph suggests that Wilson's gains at the fourth grade did 
not differ over the two years of the program and were clearly 
lower than Cherry Hill's and the state average. 



15 



10 



GAINS 



Grade 4 Reading 



WllftM 

19*0-41 



Will* Ownr Hill Avtrtgf 
1979-10 1110-Sl 13*041 



1* 
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SECTION THREE 
INVESTIGATING THE RESULTS 



So fer .you have reviewed wh^t an NCE gain means t and bow to imteipret it 
V by comparisons -t*b other standards.' The .purpdse \>f this third section is 
^o disc?uss the \next steps that you might waot feo consider. Before' 
discussing these steps, ^it is necessary* to set the stage for trie 
• ' discussion^ by describing a very simple method* for investigating a 
. A ' Title I ^project. \ - \^_^ 

. * * * * 

The Model s In describing 'complex activities such as a Title I project, 
it is often useful 4 to develop a general conceptual scheme that will help 
focus attention to relevant characteristics of 'the prdjept. ( For this 
section, you will be refining to the very simple rftodel in the following 
& figure: , % • * y " 



VJfrat You Had 



What You^id 



3. 



What iou Fouad 



scribed as 



follows: 



Very.bWefly the three canp<^ne^ts^t^e mgdel ar^p 

1. flfiat you 'hVtd ' ^ t * ■ < * % 

♦ . Thifc*category concentrates on % deScribing"what you started wrfzfi befor^ 
- " • the. project actually began.*! This' includes description^ of /the 

individuals (students and instructors) , environment (schdol, , n 
, " coramuaity^nd class room) # , and other Circumstances >uoh a's support or 
resources % ^liable. : fbe^f sctors^ll have some ef fect^bn^ the 



tpact Of your program. 
What yofr dig 



? "Xitie I instruction", is a very generic ferm that incorporates a wide 

variety* of. materials, a'nd techniques. This category concentrates on 
' describing*^ more detail what actually occurred during tfie project. t 
> This project description will help you 'focus on the critical features 
of your, project. . « \ 4 , ^ 

+ What Vou - found * ♦ 



Until now this guide has concentrated ^n orie "indication of a 
project.' s impact—the change of NCEs from pretest, to posttest' based 
off one* ach^"* rnort, '^ Mt score. i Althouah fcfiis gives a general pitta 
of a proj 
irtfor 



shievement^test score. ^ Although tfiis^ gives a general piictare 
ject's impact, you may need^to obtain more fletailed^ ^ 
^f^iot^abdbt the* impact. ^ • ^ jp! 

'.The final element. of tttiS'Trfcdel is the connection' between "What 'You' 4 - 
'Had^iand "Wfc£t You Did." Tfii-s, connection is meant *t<^ emphasize the 
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these individually affecting the results of 
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i a project, the match between the two' is also important. For example/ 
\^ ^ project could be an excellent project fbr some'students but not for 
others because of differences in their backgrounds. 



The outline p*f the section. , 

The third section follows an outline of investigation. w » 



Step 1 ; * Should there be a next step 



^Investigating the .results of a project can be time consuming. . before 
finding- and allotting the resources necessary for the' investigation/ you 
should carefully analyze' whether it is worth it. Is there interest? 
Will thdfc information be useful? Is there a potential for changing the 
project? \ 

Step 2 ; Substantiating "What You Found" * 

Jumping into a thorough investigation based on one. piece of information* 
an NCE gain, is questionable. Before embarking on an investigation, you 
sijpuld collect substantiating evictee for the "NCE gain. 

this discussion offers some alternatives that also reflect the impact of 
the TitJ.e I project and hopefully verify the observed NCE gains* 

Step 3 ; focusing the investigation. (Defining evaluation questions) * 

Deciding whether to start wit^h contextual variable^ ("W/at You Had") 6r 
implementation variables ("What You Did") can be conf^ing.^ A f ew^ ' ' 
suggestions -are offered that, might help you decide*. 

. * . . -» . r, ■ \ 

Step 4a : Investigating "wttfct 'You Had" — (Needs Assessment) \ 

Step 4b : Investigating n What You Did" — (Implementation Evaluation) 

These two discussions suggest some kinds of information that you might 

collect to better understand why your Titlfc I project had the impact it 

did. ' ' / * . 

« / • 

4 



r 



r * ♦ - - 



\ 



$ 1 ^ 



n 6 



v • : ... 



/ 
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Step 1: Should there be a next step ? 

^ Now that 'you understand NCE ✓gains better you must decide whether some 

actions should be" taken. In Title I evaluations the information can lead 
^ to three possible situations: , 

* 1. . You could- find that the Title I project produces very small or 

even negative .NCE gains, NCE gains ^could be satisfactory Dut 
* not up to expectations. You may now want to modify the existing 
program to achieve 1 higher gains or decide if you should abandon 
v the approach and adapt or. adopt ar\bther Title I instructional 

approach. 1 

\ 2. The NCE gains .from^the Title I project can be satisfactory^ to 
everyone involved-.* Resources and energies can be pUt into 
continuing the program as it is; Although a thorough 
investigation may" 6e unnecessary, you will still proDably be 
collecting some information for program nranageraent and 
implementation reasons. * 

,-3*.~ You may find that- the NGE~gairfls are- very h-igh compa^^-fco-w-bat, - 
was expected or that achieved by other projects. In this case 
you may be interested in identifying the particular strengths of 
the project either for modifying other projects in the district* 
or for disseminating the project to other schools. ♦* 

' The first and third situation, although stimulated by opposite extremes 
of results, demand that some action b'e taken: In the first situation, 
you are concerned that the ptfaject get up to par; in' the second, as an 
educator you are concerned with sharing whlfi? works well. The second' 
* ■ situation requires no f urtheiT action. It would be naive to avoid 

understanding why a*project works, however. It would be wise, to collect 
and document what information exists in case it* needs to be used in the 
future. Therefore, the answer to the question of whether there stou±o be 
? next step is Usually yes. The three situations ciiffer in how much 
information is needed and how soon. * 

% . * 



. \ 



ER1C'\ 



i7n 




21 



\ 85 



0676D 
DRAFT 



Step 2: Substantiating "What\You Found" 



( 



What You Had 




What You Did 




"Look before you leap" is a good motto for a Title' I evaiu^tor. An NCE 
gain is only one piece of information from one achievement test or even 
subtest. Now you should be able to see why it is so important that the 
test matches the pro}ect ! s curriculum, is sensitive to the projects 
effect, and is administered properly and according to the evaluation 
model's guidelines. Violations of these assumptions could result in * 
extremely negative or positive rgsult£\pr false satisfaction from an 
"acceotable" NCE gain. V ^ 



3efore you decide to proselytize a project or drastically alter it, 
verify the evaluation results. Although this certainly means checking to 
see that everything was done properly in the evaluation, you should also 
look for substantiating evidence of the project's results. If otber 
information substantiates the NCE gain then you can be fairly confident, 
about you-r decision tc? continue to investigate causes. If, on the other\ 
hand, the different sources of infotmation produce conflicting 
information, then you must investigate these conflicts first. 
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1. Other Standardized Tests 



Many Title I students take other tests outside the Title I project. For 
instance, tbe school district may administer a nationally standardized or 
locally developed test to all students. Although the test for the 
Title I students is chosen because it most closely matches the 
instruction, the results from other testing .s hould not conflict 
drastically. ' 

2. Teacher-Made Tests and Student Skills Checklists 

During the project, the Title I teacher probably administers small formal 
or informal tests to keep a record of the student's ^Pogress. Comparing 
the record of the achievement on these tests will aid in substantiating 
the NCE gains. 

3. Standardized Curriculum Tests 

If- a project has adopted a curriculum from either a large publishing firm 
or another project that is disseminating its materials, progress tests 
are often a part of the curricular materials. Often manuals are provided 
on how to interpret results from these tests. These can be used in the 
sarae-way that teacher-made tests are used to verify an NCE gain. 

4. Regular Classroom Test Results 

The Title I evaluator should also be *ware of the" achievement results of 
the Title I student in the regular classroom. T„hese tests rpay be^ more 
difficult than those given in the Title * instruction and often are given 
in more stressful situations than the Title I tests but they can be very 
useful as substantiating evidence. 

5. Gut Feelings 

Title I evaluators often concentrate ©n collecting objective information 
which is typically in the form of test results. Often there is nyjch to ■ 
be gained by documenting what the teachers and students feel -aoout the 
impact of the Title I project. This information may even identify some 
additional benefits (or detriments)' of the Title I project that ^ere not 
originally thought of to be a part of the program. Care should be taken 
in trying to provide some structure to the format of collecting * 
subjective information.' That is, just a "What do you think of .the 
Title I project?? 1 will elicit a wide range 6f responses. Some could be 
useful; some not. If there are specific areas you are interested in, 
then* address these in your collection of ^responses. 

• " „ * 

Tp expect one^hundred percent agreement from all sources is unrealistic. 
Conflicting results can, be caused by some sources not being £s exact as 

•-others or by one source keying in* on a factor that other sources ignore. 
These conflicts in themselves* can .offer information about ribw a project 
functions. You will probably ffel more certain in your investigations if 
ydu start out with most of the evidence substantiating either\egative or 

% positive results. . , ^ 
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Step 3: Focusing the investigation — (Defining evaluation questions) 

After you have decided you want tq investigate the reas6ns behind the 
gains the project' showed, you mustr decide where to. begin. Earlier we 
referred to a model: . . ; 



«$hat Ydujiad 



What You Did 



What You Found 



If you're 
There are 
the other 



limited in resources/ you* rate's t decide where to conceptrate. 
a couple reasons why you would concentrate on one area before 



1, How well do you know the Eerritory? — (Needs assessments 



The "What You Had" component in the model 
variables behind your project — what fcfnds 
training have the instructors had, for in 
this are may show that the problem is not 
that it doesn't raa^ch the students or the 
used, in evaluation the process of descr 
the strengths upon which to build and the 
- a "needs assessment.? *h thorough needs a 
success of any project. Investigations 1 
usually identify areas f;or changes in the 
variables -that affect the success. 



includes most of the contextual 
of studg^ts are served, or what 
stance.' An • investigation* of , 
that the project is poor, but 
situation-xin which it was 
ibing a situation and assessing 
weaknesses to correct' is called 
ssessment is- essential to the & 
n the ar£a "What You Had" do not 
project, but tney can identify 




Are you concerned with adjusting the project? — (Program • 
implemerttat ion) 

— ' ~ < > 

If you have a thorough understanding of "What You -Had* tjien you snould 
investigate the area called "What You Did*" Investigations here are 
referred to as "Implementation Aaluat ions"- hnd study how tne program 
actually ran (as opposed'to hot^t was designed to ,run) . The concern 
here is what refinements to the project carr be made to improve tne 
results. For 'example, yotLmay.bfc interested in how mucn actual v • 
instruction tin*e*each chriil received -or if the Title I instruction was 
coordinated with the instruction received in the regular classroom; 
These fcinds of variables are most apt jto be changed from'year to year as 
a Title I project' is refined. m ^ 

The final two part-s'of this 1 section suggest some general questicJns to 
answer in getting Co tyjderstand your project better . They areJbY no 
means meant to be exhaustive but are to help you in deciding wh^t ar.eas 
of your program you mig^ft start to investigate. Once yo^ have, decided, 
you should refer to other materials that more completely describe needs 
assessments or implementation evaluations. * 

■ «k 

* * jt 
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Step 4a; Investigating "What You Had"~(Needs Assessment) 



What You Ha< 




What You Did 



What You Found 



\ 



1. What was the average pretest achievement ? 

Sometimes by concentrating on NCE gains you may forget to looK at pretest 
scores as an indication of where the students began. Although Title I 
students are below average in* achievement/ you would certainly agjree that 
there are different levels of "below average, " A project that works well 
for students at the 25th percentile may ngt v be as successful for students 
who average at the lQth percentile. This is important to remember 
especially if- you re considering adopting a program that has worked well 
for anoth^T--<ii^£xAct. Y£u should consider whether there is any 
difference between th^ students! achievement levels and if that would 
effect the if fectivene^sn^^' project. 

2. How proficient are the students in English ? 

All the students entering a Title I program *may not have the same degree 
of ability to deal with basic English comprehension. Achievement in 
reading or matnematics is often adversely affected by language problems. 

3. What kind of student "turnover" is there? • . 



The success of a Title I project can be adversely affected if the Title 
student population has a "large percentage of students who ar^ moving in 
and out of the school during trie year. Some initial indication of , this 
problem can be obtained from the evaluation report by comparing 
"Participation, " the number of students who received any Title I 
instruction and "N," the number of students with both pretest and 
posttest scores. A high "Participation" with a t low "N" may be an 
indication of a program that has a high turnover in students. 



a - 



4. Do the Title I students have a high absentee rate ? 

If students' are only in school say three of five days a' week/" then the 
amount of instruction they receive is substantially reduced. This coulci 
be reflected in the evaluation of tne project's impact. 

5. Are the students motivated ? 

Title I evaluation concentrates on achievement impact^ however, whether a 
Etyjdent does well is affected by the students 1 motivations and 
self -concepts. A 
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6. How are students chosen to be m Title, I ? 

If you are comparing your program to another, you should .study ho** the 
stydents are chosen for the projects* l£ selection procedures vary, t 
projects may be designed for different types of students*. 



tne 



7, What, Kinds of demands does the proiect require of Title. I instructors? 



ruction to 
'eacn student. 



These demands may range, from special training in reroed 
the amount 1 of t-ifiJS^equired by the project to be. sjent 

3^ Bow would yoa describe, the school environment ? 

Schools differ on tne typs*s of studehts served, instructional sbaff 
expertise, or administrative and classroom teacher support fo* the - 
project, * ^ 

9 § How would you descrioe th,e community that the* school serves ? * 

Besides the school you may also need to describe the coramuni&y and the 
parents to identify ways in which they indirectly or airectly influence 
the Title I project* 



/ 
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ep 4b: Investigating "What You Did"? — (Implementation Evaluation) 



What You Had 



What You Did 





r 




What You Fouri£ 





1 . What general information is available about the project's 
• Implementation ? 



From your* records you can get*so$e wery general information about hotS^the 
program was implemented, fhis includes: project setting, hours per week 
of instruction for the typical student, the project length in weeks, and 
the student-);o-instructor ratio.' These ra^y be different from how the 
program was originally designed. 

*• 

2 . What specif ics -about the projects curriculum are important ? - 

• How does th£ Title I program complement th£ regulat classroom 
worK?> Is there a coordination between the two? 

• What specific materials are usecTm the program? 

• What are the project's goals and "objective s? 

How many are ttfere? 

How many general areas »a re covered? * 

At what level are they Written? (How specific are they?) 

. * i - 

• * How were the instructors .trained? 

Topics of inservice training 
* ♦ Number of days of inservice training 

• Wha'i staff levels and PTE levels are required? 

ft ' 

3. How was the project actually implemented ? . s 



1/81 



Are there specified structured teacner behaviors for the project 
'(positive reinforcements, specif ic Questioning techniques, etc*): 

Ar£ there distinctive program features such, as vplunteers, peer 
tutoring , r home' visitation/ parental- involvement? 

Where does tfce instruction'occur (in*a special room, in the 
cfassroora) ? ♦ * * 

Do the procedures complement, the objectives of tne project? 

How do teachers wor* with the* students (xadividually / large 
group, small group, combination) ? 

1 . - 0676D 
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m How is student progress monitored? 
> 

• When does instruction occur (morning, afternoon, near recess, 
6> during a particular course)? ' 



4 . How much actual instruction does a Title I student receive ? 

• * How much student time is spent in Title I per day and week? 

'# How many weeks is the student usually in Title I? 

?at are th^ criteria for exit from the program? 

/frtiati are the students missing in regular instruction? 
/ 

How much time is spsnt in testing, getting to and from the 
Title I classroom, ip administrative detils, etc.? 



* 
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EPILOGUE 



Is that all there is to interpreting evaluation restrlts? The answer is, 
of course, "No." This guide has taken you through some suggestions on „ 
how to mtffce NCE gains more meaningful and how to interpret \the gains you 
get; but, yog, can certainly expand the effects of a Title I program beyond 
achievement test results. For instance, there is the effect of the 
program on the students 1 attitudes to school an3 about themselves and 
others. Besides the students, the program may also affect others such as 
non-Title I students, teachers; administrators, or parents. 



This guide will, however, stop here and acknowledge the existence' of 
interpretation beyond this level. This rn no way should be mistaken as a 
sign that these effects are not important. In fact, they are the very 
essence of a comprehensive evaluation p4an. ' 
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TECHNICAL FOOTNOTES 



1. The effect of sampling error or gains can be estimated us€ng the 
standard error of a difference. The standard error is the expected 
^t'a.ndard deviation of gains if a large number of samples were drawn 
from the national population of Title I students. The values irfthe 
graph represent the standard error of a difference (gain) for varying 
class sizes • The formula used was: - 



SE..„ 
dif f 



2 2 
s + s - 2r s s 
x y xy x y 



N - 1 



where ' 1 

SE . « = standard error of a gain/difference 
dirf 

s 2 = variance^of pretest scores 
x 

"? • 
s = variance of posttest scores 

y 

r - = correlation between pre- and posttest scores 
xy 

\\ = number of students 

(Horst, Talbnadge and Wood) 

I ' 

Since standard deviations and correlations are generally not 
available for Title I evaluations, estimates of 16 NCEs and .50 
respectively , were made based on the data fQr a number of 
Title r project^ ' . ^ 
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2. The confidence band corresponds to a 68 percent confidence interval 
since the tabled values represent bne standard error. TfcjLs' level of 
confidence seemed appropriate since it^increased t#e chance of 
identifying small gains that wouldjiave been missed with a 95 percent, 
confidence band constructed from acout two, standard errors. This 
procedure is ^comparable to^that recommended by test publishers for 
interpreting a student's score or a group average. The publisher ^ 0 
provides the standard error of measurement for the test dnd^he 
teacher constructs a confidence band around the Score or grpup 
average. 

3. Technically, the standard error of a difference between gains should 
be used to estimate the effects of errof "in comparing gains. The 
formula is: 



SE 



diff 



St* + SE^ 



This would be complicated and difficult to estimate. It can oe 4 
shown, however, 1 tnat the sum of the two standard errors (SE]+sE2) 
is greater than one standard error of a difference. (SE diff) and less 
than two. Thus, combining the standard errors from the two gains 
results in a band that is between the 68 percept and 9 5 percent 
confidence intervals ajx2 seems* to provide a good approxiraatipn tb the 
proper statistic. ^ ^ % 

Graphically, one simply constructs th^ two confidence bands fop th& 
gains separately. If the bands overlap, the difference between the 
gains is not considered significant, this i£ essentially the same 
procedure reconunended by test publishers for interpreting student^or 
gro^> profiled, but using the standard errors for gains 'rafcher than 
the standard error of measurement. 




